OGC RBT Follow-up 2

Last week, I wrote about the Releasable Basemap Tiles Sprint and how tiles were used as an enabling technology to efficiently deliver feature data in a GeoPackage. Here, I will focus on the other half of the problem, how to visualize the feature data once it has been delivered from the GeoPackage to the client. Proper portrayal requires both styles (how lines, polygons, and text are drawn) and symbols (reusable graphic elements to be used for points and fills). We have two basic challenges[1]:

1. Architecture – we want to retain flexibility. In some environments, the styles and symbols must be delivered with the data so that the system can operate in a Disconnected, Intermittent, or Limited (DIL) environment. In other cases, the styles and symbols will be provided separately by another system. A naming scheme must allow clients to use either architecture seamlessly.
2. Interoperability – there is no perfect standard for conveying styles or symbols. They all have strengths and weaknesses[2]. We want to support commonly used standards and specifications in a way that is discoverable by clients.

The RBT Engineering Report[3] includes a section that includes a draft styling specification. As with the tiled feature data approach, the approach here divides requirements into conformance classes. The engineering report presents three conformance classes.

1. The base class establishes the schemas for four new tables needed to describe the content.
2. A second class is for fonts, which need their own table.
3. An additional class is for Mapbox GL, the styling standard used in the code sprint.

There is a lot of power in the work we have done. Sprite sheets can be used to deliver multiple symbols in the same file. The URI mechanism will allow styles and symbols to be packaged with the GeoPackage or through another system. We do have some additional work to do before we are ready to adopt this work as a standard beyond the administrative work of standards-writing:

1. A conceptual model for styling and symbology metadata, something for our tables to map to that will encourage interoperability with other emerging standards such as OGC API – Styles.
2. Additional conformance classes for other styling formats, including SLD/SE, CartoSym CSS, and CartoSym JSON, or at least an interoperable way to express which formats are being used.

[1] See https://geopackage.blogspot.com/2018/06/where-we-are-with-feature-styling-and.html for further discussion on this topic.

[2] Obligatory XKCD reference.

[3] The draft is here but a final version will be added here upon publication.

OGC RBT Sprint Follow-up 1

OGC has recently completed the Releasable Basemap Tiles Sprint. From my perspective as the GeoPackage Standards Working Group Chair, the goals of this code sprint were the following:

• Demonstrate the viability and interoperability of raster and vector map tiles in a GeoPackage;
• Exercise and review the existing specifications and draft standards for tiled feature data and styling and symbology that have been under development since at least 2019; and
• Prepare draft standards for public review and hopefully OGC adoption.

The sprint looked at three emerging standards: tiled feature data, semantic annotations, and styling and symbology. We will focus on tiled feature data here. As described in my previous post, the goal is to store feature data in a more efficient manner by leveraging the existing tiles option. This approach builds on the one devised by Mapbox in their Vector Tiles, but without being tethered to it. Mapbox Vector Tiles is Mapbox's intellectual property and the OGC community has asked for an alternative that offers more flexibility.

The approach that is in progress now and should be released soon for public comment will have  a set of conformance classes in a single document. From a governance perspective, the conformance classes allow implementers to implement as much or as little as they see fit. From a GeoPackage perspective, the conformance classes will appear as rows to the gpkg_extensions table, to make them discoverable by clients. Following are the planned conformance classes:

1. Base, establishing the gpkgext_vt_layers and gpkgext_vt_fields tables that govern what layers and what fields are in what tile sets.
2. Mapbox Vector Tiles, indicating that individual tiles are encoded using Mapbox's protocol buffers-based encoding.
3. GeoJSON, indiating that individual tiles are encoded as a GeoJSON file.
4. WKB Collection, indicating that the individual tiles are encoded as collections of Well Known Binary BLOBs. This is a relatively new option that could obviate the need for the GeoJSON option.
5. Attributes, indicating that feature attribution will be stored separately from the tiles themselves, using the GeoPackage Attributes Option.
6. Features / Tiles Mapping, indicating that the GeoPackage Related Tables Extension will be used to track the many-to-many relationship between tiles and features, potentially improving efficiency and adding improved querying options.

GeoPackage 1.4

We are working on a minor revision to the GeoPackage Encoding Standard. We propose three substantive changes to the standard.

1. Making DATETIME more flexible. "Rules as Written," a GeoPackage DATETIME must have the format YYYY-MM-DDTHH:MM:SS.SSSZ. In practice, this has proven to be unnecessarily strict. Now we will accept anything that is compatible with ISO-8601 (or its non-ISO counterpart, RFC-3339) with a Zulu / UTC time zone.
2. Relaxing Requirement 4. When GeoPackage 1.0 was published, Requirement 4 established a distinction between "GeoPackage" (a GeoPackage with no extensions) and "Extended GeoPackage" (a GeoPackage with at least one registered extension). While there was sound reasoning for this at the time, now that the standard has been out for 9 years, extensions have proven to be essential for non-trivial operations. Meanwhile, adjudication of this requirement created test skips in the Executable Test Suite. These skips were difficult to interpret. By relaxing this requirement, we remove the confusion.
3. Changing some R-Tree spatial index triggers. We discovered that one of the triggers used to maintain the R-tree spatial index was incompatible with UPSERT statements. We corrected this problem by replacing ..._update1 with two new triggers, ..._update6 and ..._update7. By deprecating the old trigger, we make it easier for clients to determine whether the change has been applied. While we were at it, we made a similar change for the trigger that was patched as part of GeoPackage 1.2.1, deprecating ..._update3 in favor of ..._update5.

We just initiated a public comment period for this change. If all goes well, we plan to publish GeoPackage 1.4 as an adopted standard by the end of this calendar year.

Introducing Application Profiles

Bottom Line Up Front: we want GeoPackage to be interoperable, but we are not there yet. It is not easy to achieve interoperability in an ecosystem that is extensible by design. Organizations that want to be use GeoPackages from disparate sources are not having great success. We have learned important lessons from the interoperability experiments that have been conducted:

• GeoPackage has many degrees of freedom, including but not limited to extensions (e.g., SRSs, geometry types, tile matrix sets, tile formats, etc.)
• conformance to the standard is no guarantee of interoperability
• there is currently no clear way to determine whether a client will be able to fully use the information available in a particular GeoPackage

I propose to solve this problem by introducing application profiles to GeoPackage. An application profile would itemize all of the optional elements in use in the database. In terms of roadmap or capability evolution, I propose a set of three incremental capability levels.

1. Verbal/written agreement
2. Machine-readable manifests that declare what options are in use in the GeoPackage so that a GeoPackage Client can determine whether it can be fully used
3. Allowing consumers to provide a "bill of materials" with a GeoPackage production request so that the ensuing GeoPackage only uses white-listed options

While Capability Level 1 would be better than nothing, I do want to propose a specific approach for the machine-readable document. Last week I introduced the concept of metadata profiles. Now I propose a new metadata profile (metadata scope: "manifest", reference scope: "geopackage") for a JSON document that captures this information. The working version of the JSON Schema for that document is on GitLab along with a sample manifest document.

It's Time We Had A Little Talk...About Metadata

I know this is going to make people uncomfortable, but it is time to talk about this. People want to put metadata in GeoPackage. It is just a fact of life. We need to be ready and that is why we are having this discussion. 

While GeoPackage has had support for metadata since its inception, I acknowledge that there is something missing. There is currently no agreement on how metadata should be used in GeoPackage to serve any particular purpose. Someone opening a GeoPackage would have no way of recognizing that the file has any particular type of metadata in it, short of inspecting every single row in the gpkg_metadata table. No way.

I propose that we address this gap by introducing "metadata profiles" to GeoPackage. A metadata profile is an agreement on what a metadata document will look like and how it will be used in the GeoPackage. I propose to leverage the extension mechanism to express this information. This approach has two parts:

1. Introduce a new extension that defines a new extension "scope" (i.e., the gpkg_extensions.scope column) of "metadata"
2. Create an extension for each metadata profile:
• using this new "metadata" extension scope
• defining the metadata scope, standard/specification, and MIME type that uniquely identify it
• defining the reference scope ("geopackage", "table", "row", or "row/col") that it will be used for 

As part of OGC's Testbed-15, I intend to explore this concept further. Here are some profiles that we are considering:

• Styles metadata
• Dataset provenance
• Delta updates
• Manifest

I will explore these further in subsequent blog posts.

Views and GeoPackage

From the beginning, the intent of GeoPackage was to allow views to be used (instead of tables) wherever appropriate. In general, "appropriate" means user-defined tables (features, tiles, or attributes). There is some confusion here because, for example, the sample feature table or view definition table (Table 7) and sample feature table definition SQL (Annex C.4) specifically call out primary keys and the notion of primary keys for view does not exist in SQLite.

In response, the GeoPackage SWG has approved some changes designed to clarify how views should be implemented. While implementers should use primary keys wherever as possible, Requirements 150 (for features) and 151 (for attributes) specify that if the database element lacks a defined primary key (i.e., it is a view), then the first column shall be primary-key-like (i.e., be a unique integer). There is a similar note for tiles but no new requirement was needed there because the schema for tiles tables already requires the first column to be the ID.

Currently, these changes are available on the working version of the standard. The working version may get published as another GeoPackage revision (1.2.2 or 1.3.0) but there is currently no timetable for doing so. One thing that we would need to do is update the Executable Test Suite (ETS) to have tests. (These tests would only apply to GeoPackages with a version after 1.2.1.) As always, if you have a GeoPackage that fails the ETS and you believe the ETS is in error, please let us know. 


P.S. Some of you in the community have asked about updatable views. It is theoretically possible to do this with a combination of tables and triggers but I don't recommend it. 

The Vector Tiles Pilot

Last year, I posted regarding vector tiles. I am pleased to announce that I have much more to report on this front. For the last six months, OGC has been sponsoring the Vector Tiles Pilot (VTP). The purpose of the VTP was to investigate how vector tiles (or tiled feature data, if you will) in the Mapbox Vector Tiles (MVT) and GeoJSON formats can be supported through the OGC standards baseline, particularly Web Feature Server (WFS), Web Map Tile Server (WMTS), and GeoPackage (GPKG). 

The initial effort concluded late last year and culminated in some videos and the following Engineering Reports (ERs):

• OGC Vector Tiles Pilot: Summary Engineering Report
• OGC Vector Tiles Pilot: GeoPackage 1.2 Vector Tiles Extensions Engineering Report
• OGC Vector Tiles Pilot: Tiled Feature Data Conceptual Model Engineering Report
• OGC Vector Tiles Pilot: WMTS Vector Tiles Extension Engineering Report
• OGC Vector Tiles Pilot: WFS 3.0 Vector Tiles Extension Engineering Report

Following the completion of the VTP, the sponsors funded an extension to the pilot (the Vector Tiles Pilot Extension or VTPExt) to take a closer look at styling of the ensuing feature data using the Mapbox Styles and Styled Layer Descriptor (SLD) encodings. This effort concluded last week and the resulting videos have been posted to YouTube. The ER is under review now (OGC members can access the current draft) and should be published in the next month or two.

From the GeoPackage perspective, the Vector Tiles Pilot allowed us to try out a series of GeoPackage Extensions:

• Tiled Feature Data: allowing tiled feature to be stored using the tiles mechanism
• Mapbox Vector Tiles: allowing tiles to contain MVT
• GeoJSON Vector Tiles: allowing tiles to contain GeoJSON
• Styles: allowing styles documents (e.g., Mapbox Styles, SLD, or others) to be stored in a way that is loosely coupled with feature data (tiled or otherwise)
• OWS Context: allowing for OWS Context information to be stored (while we have an agreement in principle here, we did not have time to test it out)
• Vector Tiles Attributes: allowing for attributes to be stored in attributes tables instead of in the vector tiles to support better querying (this topic is very complex and we did not actually reach a consensus here)

While these extensions were introduced in the ER linked above, they were further refined during the VTPExt. It is on my TODO list to add them to the GeoPackage Extensions Page. The next step is to present this information to the GeoPackage Standards Working Group (SWG) and to propose the extensions as candidate standards. I believe we have a clear use case and a sound technical approach. If the SWG agrees and there is a commitment to implementation, we may have an adopted standard later this year.